Use structured JSON output for local model classification#23
Conversation
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…regex fallback Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ation output Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…" string match - response_format is now passed by callers (callLocalInference) rather than hardcoded in generate(), so non-classification callers like suggestAnnoyingReasons still get freeform text output - Treat the literal string "null" as no-match in JSON parsing - Add test for "null" string edge case Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
| // Not JSON — fall through to regex parsing | ||
| } | ||
|
|
||
| // Fallback: freeform text parsing (backward compatibility) |
There was a problem hiding this comment.
what is this backward compatibility you speak of?
| You will be provided with a post (<post>) and a list of filter categories (<filter_categories>). | ||
| Assess whether the topic of the post relates to any of the topics in the filter categories list. | ||
| Your reasoning must be AT MOST 15 words, and MUST end with a statement of "Matches <topic>" or "No match". | ||
| Respond with JSON: {"reasoning": "<10-15 words about what the post is about>", "match": "<matched category or null>"} |
There was a problem hiding this comment.
what evals did you perform to validate this achieves similar metrics? What's the F1 score, accuracy, precision, etc...?
There was a problem hiding this comment.
this is the key point - we experimented previously with structured output like this and found that this simple of a prompt actually leads to far worse classification performance, which is why local models now receive approximately the same prompt as API ones. I'd love to be able to use a much shorter prompt if it would actually work as well since it would certainly process much faster.
The 'Check upstream' example implied local models always use a reasoning prompt (per PR imbue-ai#23). That was Qwen-era: after migrating local inference to LiteRT/Gemma, upstream itself adopted the terse table_yesno prompt. Note the lesson is model-specific + eval-gated, and that PR imbue-ai#23 was a third-party, unmerged JSON-output PR on the old codebase.
|
We've moved away from WebLLM and adopted a different form of structured output. Thank you for your early help! |
4 participants
Summary
response_format: { type: 'json_object' }to force local models to output valid JSON instead of freeform textparseLocalModelResponse()to try JSON parsing first, falling back to the existing regex-based parsing for backward compatibilityLOCAL_SYSTEM_PROMPTfrom 21 lines to 4 — format constraints are now enforced by the engine, not by prompt instructionsresponseFormatas an optional parameter togenerate()so non-classification callers (e.g. suggestAnnoyingReasons) still get freeform textProblem: Local model responses are parsed with regex looking for "Matches X" or "No match" in freeform text. When the model phrases things differently, the regex misses and the post silently gets classified as "no match" — a false negative.
Fix: Constrained decoding guarantees valid JSON output. The model can only produce
{"reasoning": "...", "match": "..."}. Zero parsing ambiguity. The vendored WebLLM (0.2.82-custom) already supportsjson_objectresponse format — it just wasn't being used.Changes
src/background/local-model.tsCLASSIFICATION_RESPONSE_FORMATconstant, makeresponseFormatoptional ongenerate(), pass it fromcallLocalInference, guard against"null"string matchsrc/shared/prompts.tsLOCAL_SYSTEM_PROMPT(21 lines → 4 lines)tests/background/local-model.test.tsTest plan
response_formatonly applied to classification calls, not suggestAnnoyingReasons